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1 Introduction 

The desire and ability to fly Unmanned Aircraft Systems (UAS) in the National Airspace System (NAS) is 
of increasing urgency. The application of unmanned aircraft to perform national security, defense, 
scientific, and emergency management are driving the critical need for less restrictive access by UAS to the 
NAS . UAS represent a new capability that will provide a variety of services in the government (public) and 
commercial (civil) aviation sectors. The growth of this potential industry has not yet been realized due to 
the lack of a common understanding of what is required to safely operate UAS in the NAS . 

NASA’s UAS Integration in the NAS Project is conducting research in the areas of Separation 
Assurance/Sense and Avoid Interoperability (SSI), Human Systems Integration (HSI), and Communication 
to support reducing the barriers of UAS access to the NAS . This research was broken into two research 
themes namely, UAS Integration and Test Infrastructure. UAS Integration focuses on airspace integration 
procedures and performance standards to enable UAS integration in the air transportation system, covering 
Sense and Avoid (SAA) performance standards, command and control performance standards, and human 
systems integration. The focus of the Test Infrastructure theme was to enable development and validation 
of airspace integration procedures and performance standards, including the execution of integrated test and 
evaluation. In support of the integrated test and evaluation efforts, the Project developed an adaptable, 
scalable, and schedulable relevant test environment incorporating live, virtual, and constructive elements 
capable of validating concepts and technologies for unmanned aircraft systems to safely operate in the 
NAS. 

To accomplish this task, the Project planned to conduct three integrated events: a Human-in-the-Loop 
simulation and two Flight Test series that integrated key concepts, technologies and/or procedures in a 
relevant air traffic environment. Each of the integrated events were built on the technical achievements, 
fidelity and complexity of previous simulations and tests, resulting in research findings that support the 
development of regulations governing the access of UAS into the NAS. 

The purpose of this document is to describe how well the system under test was representative of a 
“relevant environment”, ensuring the research objectives outlined in the Integrated Human-in-the-Loop 
(IHITL) Simulation Test Plan (IHITL SIM TP-01) were met. Section 1 provides background, high-level 
objectives and a description of the simulation components. The simulation had six high-level objectives, 
which required three different test setups. The description of these setups is detailed in Section 2. This 
section also contains a summary of the data gathered including any anomalies reported during collection. 
Section 3 discusses lessons learned during the conduct of the simulation. Section 4 provides test 
environment analysis, including details on system latency as well as feedback from test subjects and subject 
matter experts. 

1.1 General 

The integrated IHITL simulation provided data to the UAS researchers to evaluate the state of the 
simulation environment development by integrating and testing key Sense and Avoid (referred to as Detect 
and Avoid in the RTCA SC 228 ToR) technologies into a research Ground Control Station (GCS). The 
technical goals for the IHITL were to: 1) evaluate and measure the effectiveness and acceptability of SAA 
systems (algorithms and displays) to inform and advise UAS pilots; and 2) evaluate and measure the 
interoperability and operational acceptability of UAS integration concepts for operating in the NAS. A third 
Project goal was to characterize the simulation and test environment in order to evaluate the state of the 
simulation architecture with respect to future UAS research activities. The completion of the IHITL 
provided valuable data to the SSI and HSI research as well as reduced the risks associated with building a 
relevant test environment moving towards the Project flight tests. 

NASA Ames, NASA Armstrong, and NASA Langley Research Centers shared responsibility for 
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conducting the tests, each providing a test lab and critical functionality. The experiment was divided into 
three distinct test setups each focusing on different aspects of the primary technical goals. The first was a 
three- week study looking at the controller acceptability of pilot initiated maneuvers based on SAA alerts, 
resolutions, and the GCS displays. The second was a two- week study focusing on the acceptability of the 
same SAA alerts, resolutions, and GCS displays to the UAS pilot. The third was divided into two test 
conditions, the first investigated the threshold between the existing Traffic Alert and Collision Avoidance 
(TCAS) II alerting logic present on commercial aircraft and potential self- separation alerting for UAS; the 
second investigated aspects of UAS communication delay and wind uncertainty with respect to Air Traffic 
Control (ATC) and UAS pilot maneuver negotiation. 

The three test setups were conducted in parallel and divided by the Center participants. The first and second 
setups, which had similar hardware and software requirements, were conducted out of NASA Ames and 
NASA Armstrong over a seven- week period beginning 9 June 2014. The third test setup test condition 
involving the TCAS II alerting logic, run jointly between NASA Ames and NASA Langley’s UAS Lab, 
started data collection on 9 June 2014 over a three- week period. The third set setup test condition involving 
communication and wind uncertainty was run standalone out of the NASA Langley’s UAS Lab from 9 
June 2014 to 9 July 2014. Over the course of the IHITL, data was collected from a total of 10 pilot and 21 
controller subjects from the combined three test setups. 

Test facilities fell into two categories: 

Development Facilities: 

• Distributed System Research Laboratory (DSRL) at NASA Ames 

• Flight Deck Display Research Laboratory (FDDRL) at NASA Ames 

• LVC Distributed Environment Lab at NASA Armstrong 

• Research Ground Control Station (RGCS)/UAV Simulation Development Lab at NASA 
Armstrong 

• UAS Sense and Avoid Research (USAR) Lab at NASA Langley (Stinger Gaffarian Technologies 
(SGT)) 


Test Facilities: 

• Crew Vehicle Simulation Research Facility (CVSRF) at NASA Ames 

• Distributed System Research Laboratory (DSRL) at NASA Ames 

• LVC Distributed Environment Lab at NASA Armstrong 

• RGCS/UAV Simulation Development Lab at NASA Armstrong 

• UAS Sense and Avoid Research Lab at NASA Langley 


1 .2 Background 

The UAS Integration in the NAS Project has ongoing research efforts focusing on the investigation of 
Sense and Avoid, defined by the collision avoidance (CA) and self separation (SS) concepts, and its 
interoperability with separation assurance (SA). As shown in Figure 1 , these concepts have distinct 
thresholds (which are a focus of the research) that may overlap in both temporal and spatial domains. 

The self- separation algorithm research was coupled with the investigation of the UAS GCS pilot 
environment, display of the SAA advisories to the pilot, and interaction between pilots using these 
advisories and air traffic controllers. As such, the Project conducted a series of integrated human-in- the- 
loop simulations in order to evaluate pilot and controller acceptance of the usability, display, and timeliness 
of different types of SAA advisories. The IHITL utilized the Distributed Test Environment (DTE) 
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developed by the Project to provide the core infrastructure and supporting simulation Live, Virtual, 
Constructive (LVC) software components in order to simulate a virtual UAS flying under nominal (non- 
contingency) operations, interacting with ATC and virtual manned aircraft. An instance of the DTE was 
configured to meet the requirements for each of the IHITL test setups, providing the appropriate level of 
functionality, fidelity, and security. The DTE test components included a research prototype GCS, an 
engineering GCS display, a flight simulator with an integrated TCAS system, and constructive aircraft 
target generators and virtual ATC workstations. 


Collision Avoidance Self Separation Sense-and-Avoid 
Threshold Threshold Threshold 



Separation Assurance/ ATC Coordination 




Sense and Avoid 


I 



SS Alert 


Preventive Alert 


Conflict Alert 


Avoicfance Self Separation Controlled Separation 


Figure 1. Sense and Avoid and Separation Assurance Interoperability. The Sense and Avoid concept 
encompasses the collision avoidance, self-separation, and boundary with separation assurance. 


1 .3 Overall Test Goals and Objectives 

The integrated Human in the Loop simulation served as the mechanism to test two primary technical goals: 

1 . ) Integrate and evaluate the state of UAS concepts and supporting technologies defined within the 

scope of the UAS Integration in the NAS Project. Identify areas of future research and 
development emphasis and reduce risk for the flight tests 

2. ) Evaluate and measure the effectiveness and acceptability of the SAA algorithms and displays to 

inform and advise pilots of UAS aircraft, as well as the acceptability of resulting maneuvers and 
workload to the air traffic controller. 

These high-level test goals were further broken down into six general test objectives: 

• Integrate and evaluate the state of UAS concepts and supporting technologies defined within the 
scope of the UAS Integration in the NAS Project as a function of traffic scenarios 

• Evaluate the pilot and controller acceptability of UAS maneuvers in response to SAA advisories 

• Assess the impact of variable winds on the execution of SAA advisories as well as interoperability 
with TCAS equipped aircraft 
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• Assess the impact of communication delay and wind uncertainty on the air traffic controller 
workload and acceptability of SAA self- separation maneuvers 

• Evaluate the acceptability and performance of the LVC (i.e. software components of the DTE) to 
provide a relevant environment 

• Collect data to improve batch simulation models. 


1.4 Simulation Configurations 

The IHITL has two distinct system configurations shown in Figure 2 and Figure 3. These figures describe 
the system level requirements for IHITL in the abstract, the specific hardware and software components 
that comprise the implemented test setups are described later in this document. 

Figure 2 provides a high-level diagram of IHITL System Configuration 1. System Configuration 1 was 
designed to connect virtual air traffic control (ATC) and constructive aircraft processes running at NASA 
Ames with a prototype research UAS GCS at NASA Armstrong. The framework for the simulation DTE 
was supplied by LVC components connected via the High Level Architecture (HLA) messaging 
infrastructure. A virtual UAS simulator was integrated into a prototype research GCS along with a traffic 
display (GCS-TD) used to present SAA (SS and CA) advisories to the pilot. The LVC components sent and 
received data through a gateway connected to the HLA network. The constructive manned aircraft and 
ATC workstations communicated directly via a local gateway and communicated to the other components 
via that gateway and the HLA. The constructive manned aircraft generators provided the required 
background traffic supporting a more realistic environment. 
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NASA Ames 


Figure 2. System Configuration 1. 


Figure 3 shows a high-level diagram of the IHITL System Configuration 2. As with System Configuration 
1, the System Configuration 2 DTE was designed to provide connectivity among the LYC components, 
specifically a virtual UAS GCS, constructive manned traffic generators, and virtual ATC. In this case these 
virtual and constructive components were provided via facilities at NASA Langley. In addition, a flight 
simulator with TCAS installed ran at NASA Ames providing the primary intruder aircraft via HLA. A UAS 
GCS at NASA Langley contained a combined Virtual Simulator, GCS-TD, and SAA system (self- 
separation and CA) for testing. 
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Core LVC 
Software 


Figure 3. System Configuration 2. 


The high-level system architectures shown in Figure 2 and Figure 3 provided the basis for the specific LVC 
simulation environment that was developed for each of the three test setups conducted during the IHITL: 

1 . ) Air Traffic Controller (or Controller) Acceptability of SAA Maneuvers (Test Setup 1) 

2. ) Pilot Acceptability of SAA Maneuvers (Test Setup 2) 

3. ) Acceptability of SAA Advisories (Test Setup 3). 

It should be noted that Test Setup 3 was divided into two test conditions using subsets of the system 
configuration shown in Figure 3. The first test condition utilized the connection between NASA Ames and 
NASA Langley to connect the flight simulator with the GCS. The second condition, focusing on controller 
response to the SAA advisories, was conducted stand-alone at NASA Langley using only the components 
shown in the upper box in Figure 3. The descriptions for each test setup are further detailed in Section 2. 

2 Test Resources 

Each test setup had distinct associated LVC components interconnected into a distributed test environment 
instantiation; however, many of the LVC components (located in laboratories) are common among the 
setups. This section provides detailed background descriptions of the distributed test environment LVC 
components and other test resources. Explicit detail regarding how these resources were used during test is 
captured in the Test Methodology sections for each Test Setup. 
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2.1 High Level Architecture (HLA) and LVC Gateway 

The framework for the simulation environment was supplied via the High Level Architecture (HLA) 
messaging infrastructure. The connections used a version of the IEEE 1516 standard Pitch portable Real 
Time Infrastructure HLA and Federation Object Model (FOM) middleware to exchange information about 
the air traffic environment (aircraft state, flight plans, etc.) among the participants operating from 
distributed facilities. 1 The HLA utilized Toolboxes to convert data from simulation components (e.g. flight 
simulator, or air traffic control display) into its expected format. Messages sent to a distributed facility were 
routed to the HLA via a Gateway and corresponding Toolbox. 


2.2 Remote User Monitoring System (RUMS) 

In order to facilitate the monitoring of the data collection Remote User Monitoring System (RUMS) 
software was used. The RUMS server connected to the Gateway process and recorded all data messages to 
a remotely accessible database. Test participants were able to connect to the database via queries from an 
Internet web browser and display data of interest. 

2.3 Multi-Aircraft Control System (MACS) 

The Multi- Aircraft Control System (MACS) program was a comprehensive simulation program developed 
at the NASA Ames Airspace Operations Laboratory (AOL) to support research in air traffic operations in 
the current and NextGen environment. 11 The underlying MACS software was used extensively in the IHITL 
to provide aircraft control emulation, aircraft target generation, and a standalone UAS ground control 
station with integrated SAA algorithms. The following sections provide more detail on each of the 
functionalities. 

2.3.1 Air Traffic Control Display 

The Multi- Aircraft Control System (MACS) program provided the ATC display functionality for the IHITL 
simulation. An instance of the MACS was used for each ATC sector position. Depending on the start-up 
configuration, it provided an emulation of the En Route Automation Modernization (ERAM) environment 
or Standard Terminal Automation Replacement System (STARS) environment. Figure 4 shows MACS 
configured as an ERAM sector display. 
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Figure 4. Multi-Aircraft Control System (MACS) Air Traffic Control displays. 


2.3.2 Aircraft Target Generation 

The MACS SimMgr and MACS pseudo pilot programs provided simulated aircraft targets during testing 
(see Figure 5). At the beginning of each IHITL simulation MACS SimMgr read the initial conditions and 
flight path from an input scenario file. Aircraft were then assigned to the MACS pseudo pilot stations 
where the aircraft position updates were generated based on the flight paths and aircraft model data. This 
aircraft data was distributed to each test participant as required. MACS used a four degree-of-freedom 
trajectory engine to update the location of the aircraft on a one second frequency (emulating ADS-B). The 
simulated targets emulated either IFR or YFR aircraft and provided background traffic that flew a 
predetermined flight path. 
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Figure 5. Multi- Aircraft Control System (MACS) pseudo pilot displays. 


2.3.3 UAS GCS 

In order to support display development for the project, MACS was expanded to function as a standalone 
Pilot Station with built-in UAS characteristics providing a virtual GCS, called the MACS GCS (see Figure 
6). This version of MACS had the NASA Langley Stratway+ self- separation system integrated into its 
software. The MACS GCS was used for both test conditions for Test Setup 3 and provided the position 
updates for the primary UAS aircraft of interest in each scenario. 



Figure 6. MACS Ground Control Station displays. 
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It should be noted that the MACS program interfaced with the test environment through an associated 
gateway called the Aeronautical Data link and Radar Simulator (ADRS). The use and connectivity of the 
MACS programs to the test environment via the ADRS is shown in the connection diagrams in Section 2.1, 
when it was connected and not operated stand-alone. 

2.4 Vigilant Spirit Control Station (VSCS) 

Air Force Research Lab (AFRL)’s Vigilant Spirit Control Station (VSCS) was a UAS simulator that 
modeled the virtual flight of a UAS aircraft (See Figure 7) m . It connected to the test environment via the 
HLA, providing position updates based on flight plan and maneuvers. VSCS also had a separate traffic 
display, which provided the GCS-TD functionality. The traffic display showed self- separation and collision 
avoidance conflict advisories in addition to intruder information such as call sign (if available), relative 
altitude, vertical velocity, and ground speed. The VSCS traffic display also showed resolution maneuvers 
and supported “vector-planning”. Vector-planning allowed the pilot to test various horizontal or vertical 
vectors to help determine appropriate trajectories to avoid potential conflicts. Maneuver resolutions and 
vector-planning were facilitated by the SAA system, which was derived from the AutoResolver 
technologies developed by NASA Ames to support resolution advisories for manned aircraft. AutoResolver 
connected via a Gateway, receiving data from VSCS and MACS SimMgr, and sending advisories back to 
the VSCS and presented on the traffic display for the UAS pilot. 



Figure 7. Vigilant Spirit Control System (VSCS) Integrated Traffic and Tactical Situation Display. 

2.5 B747 Flight Simulator 

The existing B747 Flight Simulator at NASA Ames provided an additional manned virtual aircraft and 
connects to the other LVC components via the HLA. The simulator has an on board Flight Management 
System (FMS), which allowed for flight plans to be flown in simulation. The virtual position updates were 
generated internally, incorporating a realistic flight model and winds. The B747 Cab was equipped with the 
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TCAS II version 7.0 algorithm and was used to test interoperability between the UAS SAA algorithm and 
existing manned aircraft tactical avoidance algorithms. Data used by TCAS to determine alerts was 
automatically recorded and available for post processing by the researchers. Figure 8 shows pictures of the 
inside and outside of the B747 flight simulator. 



Figure 8. Boeing 747-400 Flight Simulator at NASA Ames. 


2.6 Research Ground Control Station (RGCS) 

The UAS Ground Control Station (GCS) capability was provided by the RGCS at NASA Armstrong for 
Test Setups 1 and 2. The RGCS was a hardware test-bed for UAS GCS information display and human 
factors concepts. It contained the monitors and computer systems that run the display systems under test. 
The RGCS used the VSCS as the UAS aircraft simulator, as well as the VSCS traffic display for its GCS- 
TD instantiations. A graphical representation of the RGCS is depicted in Figure 9. 



Figure 9. Research Ground Control Station layout. 
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3 Test Setups 

The IHITL had three distinct test setups tracing back to the two system configurations shown in Figure 2 
and Figure 3 corresponding to the specific objectives, hardware, and software required to conduct the 
experiment. The first two test setups utilized the RGCS UAS pilot station developed at NASA Armstrong, 
the first with controllers (stationed at NASA Ames) as subjects and the second with UAS pilots as subjects 
The third test setup integrated a flight simulator with a TCAS II system stationed at NASA Ames into an 
existing simulation environment at NASA Langley. Distributing the simulation among the NASA Centers 
allowed the project to utilize the existing simulation infrastructure resident at those facilities. 


3.1 Test Setup 1 - Controller Acceptability of SAA Maneuvers 

The experimental goal of this study was to evaluate the controller acceptability of maneuvers performed for 
“self- separation” to remain well clear without a new clearance. System Configuration 1 provided the 
baseline architecture for this test setup, which utilizes a virtual UAS aircraft flown on an IFR flight plan 
with constructive background traffic scenarios containing a mix of manned IFR and VFR aircraft. In this 
setup, UAS and manned aircraft pilots acted as confederates and controllers were the test subjects. The 
VFR traffic included non-cooperative (non- squawking) aircraft, which allowed for the UAS to interact with 
manned aircraft in close proximity to its area of operations without ATC resolution. Encounters between 
the UAS and the manned background traffic were scripted to trigger the self- separation algorithm to send 
alerts and advisories to the GCS-TD. The UAS pilot used the advisories to coordinate with controllers to 
strategically mitigate potential conflicts. 

The Sense and Avoid system under test included a modified version of the AutoResolver, developed by 
researchers at NASA Ames . The AutoResolver was first developed to support air traffic controllers with 
advisories to maneuver aircraft in the en route and Terminal airspace based on predicted Loss of Separation 
(LOS). lv,v This algorithm was modified to work with pilots to support alerts and maneuvers in response to 
LOS predictions and included a model of an airborne sensor that applied a filter to restrict the inputs to the 
AutoResolver based on predicted airborne sensor performance (range and field of regard) . Though not a 
focus of this test, to help determine cases where the self- separation algorithm failed to properly predict and 
resolve conflicts, a collision avoidance algorithm (based on TCAS parameters) was used to alert the UAS 
pilot of impending collision. 

3.1.1 Test Methodology 

Test setup 1 utilized a virtual UAS aircraft flown under an IFR flight plan in scenarios with a mix of 
manned IFR and VFR traffic. During test setup 1 , pilots acted as confederates and controllers were the 
subjects. VFR traffic allowed for the UAS to interact with the manned aircraft, which enabled the SAA 
system to provide advisories to the pilot based on the definition of the Well Clear Violation (WCV) 
parameters. In order for “well clear” to be violated all of the following conditions must be met: 

a. The horizontal Closest Point of Approach (CPA) or the current horizontal range was less 
than 0.8 nmi 

b. The time to horizontal CPA was below 40 seconds 

c. The time to co-altitude was below 40 seconds or the current vertical separation distance is 
below 500 ft. 

The UAS pilot used the advisories to work with controllers to strategically mitigate potential conflicts. 

The ConOps for this test setup was a UAS being controlled by a GCS located in southern California, with 
the aircraft deployed under various missions in the San Francisco Bay area. In this test, traffic scenarios 
running on MACS at NASA Ames provided background traffic in Oakland Center (see section 3.1 .1 .1 for 
an airspace description). The MACS pseudo pilot stations sent position updates every second. The UAS 
was flown by an instance of VSCS running on the RGCS at NASA Armstrong following a prescribed flight 
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plan. Controllers, the subjects of these tests, managed the traffic on MACS running as an ERAM display at 
NASA Ames. The SAA system was connected to the RGCS and provided advisories to the UAS pilot who 
was able to trial plan different maneuvers via the VSCS traffic display and negotiate an appropriate 
resolution to any predicted conflict with the controller. The simulated UAS sensor suite provided position 
data for each intruder aircraft to the SAA algorithm based on its aircraft equipment and flight plan type (see 
Figure 10): 

1. ADS-B: 

a. Range 80 nm 

b. Altitude +/- 5000 ft 

2. Non-Cooperative: 

a. Range: 6 nmi 

b. Azimuth: +/- 110 degrees (from nose) 

c. Elevation: +/- 20 degrees (from horizontal) 


Altitude: 
+/- 5000’ 


{ 


Elevation: , 
+/- 20 deg 1 


L 


J 


Range: 80 nmi radius 


Azimuth: 



On Board Radar 
Emulation 

ADS-B Sensor Emulation 

Figure 10. Aircraft sensor emulation values used for Test Setups 1 and 2 (not to scale). 


Since the ConOps for the test was chosen to mirror the physical instantiation of the distributed system 
(UAS operating near central California and UAS GCS located near southern California), no additional 
communication system modeling was required to emulate the latencies. However 93ms uplink delay and 
115ms downlink delay were added to VSCS to emulate latencies associated with data message passing 
between the GCS and aircraft. No winds were modeled for this Test Setup. 

Controllers were instructed to separate traffic as they normally would, issuing traffic advisories and alerts 
to appropriate pairs of aircraft whenever separation was projected to decrease below the applicable 
minimum. Four scenarios of 40 minutes each were used, two with a high traffic load, two with a nominal 
traffic load. See Section 5 for a detailed description of the four scenarios. During each scenario, the pilot 
was instructed to either request a maneuver from the controller or maneuver without controller 
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coordination. Three “types of VFR traffic” were present in every scenario, and each encounter listed below 
was between an IFR aircraft and one of these three types of VFR aircraft: 

non-cooperative, non-participating VFR aircraft (primary targets only) 

cooperative, non-participating VFR aircraft (transponder-equipped, not talking to ATC) 

cooperative, participating VFR aircraft (transponder-equipped, talking to ATC) 


3. 1.1.1 Airspace 

Figure 1 1 shows the simulation airspace, which was an emulation of Oakland Center (ZOA) Sectors 40 and 
41 as a combined en route sector. Sector 40/41 extends to the California coast and coastal mountains in the 
west to the western Travis Air Force Base in the east. The sector overlies the Napa and Sonoma valleys 
from Mendicino to Sausalito. Sector 40/41 airspace included all classes of airspace from the surface to 
Flight Level 240. 



Figure 11. ZOA 40/41 airspace. 

3. 1.1. 2 Scenarios 

There are four explicit aircraft scenarios, which include aircraft executing specific mission profiles as well 
as transitioning to and from a major Terminal Radar Approach Control (TRACON). Each scenario 
contained scripted aircraft encounters running for approximately 40 minutes and was designed to trigger 
the SAA alerting logic in order to capture the appropriate data. 

3. 1.1. 2.1 Search and Rescue 
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This scenario simulates UAY flight operations in support of search and rescue of a group of hikers reported 
lost in the wilderness area in the Mayacmas Mountains between Clear Lake and Lake Berryessa. The 
aircraft enters the scenario at 15,000 feet and starts a descent to 7,000 feet to enter a grid search pattern. 

The UAV then simulated visual reconnaissance from 6,000 feet to 7,000 feet. Upon exiting the grid the 
UAV climbed to 17,000 feet and proceeded direct to Williams VORTAC enroute to Beale AFB. All 
operations were conducted in Class E airspace using instrument flight rules. This profile was designed to 
conform with the FAA CONOP, UAS National Plan (Initial), September 2012, 5.5 Grid Pattern. Refer to 
Table 1 and Figure 12 for a description and diagram of the UAS flight path. 


Table 1. Search and Rescue Flight Profile (Route 1) 


Point # 

Fix 

Lat/Long 

Altitude 

1 

Sim Start Point 

3826.7 1/-12217.5 1 

15,000 

2 

OZIEE 

3829.53/-12223.18 

Descend to 12,000 

3 

WUSHU 

3845.81/-12219.25 

Descend to 7,000 

4 

3 847 .03/- 12225 .23 


7,000 

5 

3830.77/-12229.92 


7,000 

6 

POPES 

3829.15/-12220.75 

Descend to 6,000 

7 

3836.53/-12218.70 


6,000 

8 

3845 .1 8/- 12216.35 


6,000 

9 

3 846 .43/- 12222 .06 


6,000 

10 

NOFTE 

3830.46/-12226.62 

6,000 

11 

STS 

3830.48/-12248.63 

Climb to 17,000 

12 

ILA 

3404.36/-12201.81 

Descend to BAB 


20 






IHITL Test Environment Report - February 17, 2015 



Figure 12. Projected flight path of UAY on the search and rescue mission in the ZOA airspace. 

3. 1.1. 2.2 CDF Fire Line 

These routes simulated a UAV as an air asset in support of fire fighting operations with the California 
Department of Forestry (CDF). In this scenario, a large fire had been burning in the Mayacmas Mountains 
north of Clear Lake. The fire line had been moving rapidly driven by strong northwesterly winds. CDF 
command needed information on existing fire lines in order to better allocate resources. The UAV entered 
the track at Flight Level 200 and immediately descended to 8,000 feet. The aircraft returned to Flight Level 
200 after transiting WUSHU intersection and remained at that altitude for the remainder of the scenario. 
Flight operations began and ended in Class A airspace. Mission altitudes and the associated encounters 
were conducted in Class E airspace. The UAV remained at the same altitude throughout the scenario to 
facilitate traffic encounters at lower altitudes. This profile was designed to conform to the FA A CONOP, 
UAS National Plan (Initial), September 2012, 5.3 Loiter for Surveillance. Refer to Table 2 and Figure 13 
for a description and diagram of the UAS flight path. 
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Table 2. CDF Fire Line Flight Profile (Route 2) 


Point # 

Fix 

Lat/Long 

Altitude 

1 

Sim Start (VICOR) 

3924.83/-12322.10 

FL200 descending to 
8,000 

2 

WEPLO 

3914.26/-12305.63 

8,000 

3 

FOWND 

3905 .76/- 12257 .80 

8,000 

4 

LAPED 

3903 .70/- 12247 .64 

8,000 

5 

WUSHU 

3845.81/-12219.25 

Climb to FL200 

6 

DALON 

3904.14/-12215.99 

FL200 


3912.1 9/- 12243 .62 


FL200 


3907.78/-12250.30 


FL200 


39 14.62/- 12252.3 1 


FL200 

7 

SABBY 

3917.53/-12302.44 

FL200 

8 

HUPTU 

3930.00/-12244.65 

Descend to BAB 



Figure 13. Projected flight path of UAY on a fire line mission in the ZOA airspace. 
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3.1. 1.2.3 USCG Coastal Watch 

In this simulation, the UAV conducted flight operations in cooperation with the USCG. The Coast Guard 
had requested that the UAV conduct routine surveillance of the beach areas from Point Reyes to Point 
Arenas. Upon arrival at Point Arenas, the UAV was requested to perform a visual search for a small craft 
near Point Arenas that had requested assistance from the USCG. The UAV flew a single orbit southeast of 
AMAKR intersection to search for the vessel then proceeded on course. The UAV entered the simulation at 
10,000 feet and descended to 8,000 feet to conduct the beach surveillance. The aircraft climbed to 14,000 
feet 10NM west of FROSH intersection to conduct the search for the small vessel. Once the surveillance 
was complete the aircraft descended to 7,000 feet en route to Santa Rosa airport. Flight operations were 
conducted in Class E airspace using instrument flight rules. This profile was designed to conform to the 
FAA CONOP, UAS National Plan (Initial), September 2012, 5.3 Loiter for Surveillance. Refer to Table 3 
and Figure 14 for a description and diagram of the UAS flight path. 

Table 3. USCG Coastal Watch Flight Profile (Route 3) 


Point # 

Fix 

Lat/Long 

Altitude 

1 

Sim Start Point 

3753.30/-12235.45 

10,000 

2 

LOZIT 

3753 .96/ - 1 2240 .40 

10,000 

3 

MOLEN 

3759.86/-12305.20 

Cross @ 8,000 

4 

BOARS 

3813.64/-12307.62 

Descend to 4,000 

5 

AMAKR 

3900 .00/- 12345 .00 

Cross @ 14,000 

6 

3902 .21/- 12340 .72 


14,000 

7 

3849 .29/- 12329 .73 


14,000 

8 

3846.89/-12334.30 


14,000 

9 

AMAKR (outbound) 

3900 .00/- 12345 .00 

Descend to 7,000 

10 

FROSH 

3836.84/-12311.74 
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Figure 14. Projected flight path of UAY on a US Coastal Watch mission in the ZOA airspace. 

3. 1.1. 2.4 Freighter 

The UAV simulated a Cessna Caravan, an aircraft contracted to Federal Express. The UAV was conducting 
a regularly scheduled flight from Redding Airport to the FedEx nexus at Oakland International Airport. 
Letters of Agreement (LOAs) between Oakland ARTCC, Northern California TRACON and FedEx 
dictated that passenger aircraft have priority over unmanned freighters. This required the UAV to execute a 
turn in holding at STIKM waypoint on the RAIDR3 STAR as the OAK airport had several passenger 
aircraft on approach. The UAV entered the simulation at 16,000 feet and descended to enter holding at 
7,000 feet. After completing the turn in holding the aircraft continues to Oakland Airport via the RAIDR4 
STAR at 7’ 000 feet. Flight operations were conducted in accordance with FAR Part 121 and in Class E 
airspace using instrument flight rules. This profile was designed to conform to the FA A CONOP, UAS 
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National Plan (Initial), September 2012, 5.6 Point-to-Point. Refer to Table 4 and Figure 15 for a description 
and diagram of the UAS flight path. 

Table 4. FedEx Freighter Flight Profile (Route 4) 


Point # 

Fix 

Lat/Long 

Altitude 

1 

Sim Start Point 

39 18. 95/- 12226 .65 

16,000 

2 

STIKM 

3832.00/-12239.00 

Cross @ 7,000 

3 

3832.87/-12232.00 


7,000 

4 

3845 .91/-12234.02 


7,000 

5 

3845 .09/- 1224 1.49 


7,000 

6 

STIKM 

3832.00/-12239.00 

7,000 

7 

RAIDR 

3801.95/-12233.45 

7,000 



Figure 15. Projected flight path of UAY on a FedEx Freighter mission in the ZOA airspace. 
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3.1.2 Test Participants 

Air traffic controller subjects were retired en route controllers, experienced with using the ERAM display. 
Controllers were trained by the test team controller SME on the specific aspects of the ZOA 40/41 sector 
aircraft flow. Non-subject pilot participants manning the pseudo pilot stations had the appropriate IFR 
experience to communicate with ATC and experience running the MACS pilot software. The UAS pilots 
(non subjects) were experienced UAS pilots with prior training on using the VSCS and the RGCS system. 

3.1 .3 Test Environment Architecture 

The focus of this study was to gather data to help determine whether the pilot is required to coordinate a 
self- separation maneuver with ATC or whether they are allowed to maneuver to remain well clear without 
a new clearance. Metrics to determine the impact of coordinating (or not) self- separation maneuvers to 
remain well clear included time alert was presented to the UAS pilot, maneuver type (vertical, horizontal), 
perceived controller workload (measured post-scenario via questionnaire), the amount of time spent 
negotiating avoidance maneuvers, and an acceptability assessment questionnaire administered to the 
controller at the end of each collection run. 

Figure 16 shows the test environment design for the Controller Acceptability of SAA Maneuvers test setup. 
The core connectivity infrastructure was provided by the HLA messaging system running at the DSRL lab 
at NASA Ames. A UAS pilot utilized the RGCS functionality at NASA Armstrong’s RGCS lab. The UAS 
simulation functionality was provided by the VSCS instantiated in the RGCS along with the VSCS traffic 
display, which provided the display of the surrounding air traffic and SAA advisories to the pilot. The SAA 
algorithm was derived from the AutoResolver technologies developed by NASA Ames to support 
resolution advisories for manned aircraft and was connected to the RGCS via a Gateway. The MACS 
SimMgr and Pseudo Pilot stations running out of the CVSRF provided virtual and constructive manned 
background and intruder aircraft. The ATC subjects utilized the MACS ATC environment, which was run 
out of the CVSRF at NASA Ames. The MACS processes communicated to each other and the rest of the 
FVC components via the ADRS, which in turn connected to HFA. A distributed monitoring system 
consisting of an instance of MACS ATC was connected in the FVC lab at Armstrong for situation 
awareness of the traffic presented to the air traffic controllers. 
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Figure 16. IHITL LYC system with components for the Controller Acceptability of SAA Maneuvers 
Test Setup 1 (including observer positions) 


3.1.4 Simulation Voice Comm 

The IHITL configurations used a combination of Simphonics and Plexsys voice communication systems 
that utilized Distributed Interactive Simulation (DIS) IEEE 1278.1 A- 1998 v6 Standard Protocol. The 
system was comprised of hardware and software that distributed multicast packets voice over a local area 
network. Packets were routed to specific receivers based on a simulated “frequency” that each station 
monitored. At NASA Ames, test subject, confederate, researcher, test- engineering, and control room 
positions located within the CVSRF utilized Simphonics systems, and test-engineering positions located 
within the DSRL utilized PlexComm systems. Voice recording for test setup 1 were accomplished using 
the Simphonics Recorder in the CVSRF ATC Lab. Eight Plexsys PlexComm Virtual Radios were installed 
on at NASA Armstrong for use by the RGCS pilot, researchers, test engineers, and control room. 

The firewall configuration that establishes a VPN connection between Ames- Armstrong did not permit 
UDP (multicast) packets to pass through the VPN. Therefore, tunneling software that converts the UDP 
packets to TCP/IP and back to UDP on each end was installed. 
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Figure 17. IHITL LVC Voice Communication connectivity for System Configuration 1 


3.2 Test Setup 2 - Pilot Acceptability of SAA Maneuvers 

The experimental goal of this study was to continue the evaluation of candidate SAA displays and 
algorithms with respect to self-separation and collision avoidance, based on results and lessons learned 
from previous internal simulations. System Configuration 1 also provided the baseline architecture for this 
test setup, which was run with the UAS pilot as the subject. The virtual UAS aircraft was flown on an IFR 
flight plan with scenarios containing a mix of manned IFR and VFR (squawking) aircraft. In this setup, 
controllers acted as confederates, allowing for interaction between the manned and UAS aircraft. The SAA 
algorithm provided alerts and advisories for display to the pilot on the GCS-TD. The pilot used the display 
information to negotiate maneuvers to avoid the traffic with ATC. A collision avoidance algorithm was 
used under this test setup to test the SS-CA time boundary. 

As with the previous test, AutoResolver and the collision avoidance algorithms provided the SAA 
capabilities for this test set-up. 

3.2.1 Test Methodology 

The UAS pilot was the subject for this test setup. The virtual UAS aircraft flew under an IFR flight plan in 
scenarios with a mix of manned IFR and VFR traffic. With controllers as confederates, allowing for 
interaction between the manned aircraft and the UAS, the SAA system provided advisories to the pilot 
based on the definition of Well Clear Volume (WCV) parameters. In order for “well clear” to be violated 
all of the following conditions were met: 

a. The horizontal CPA or the current horizontal range was less than 0.8 nmi 

b. The time to horizontal CPA was below 40 seconds 

c. The time to co-altitude was below 40 seconds or the current vertical separation distance 
was below 500 ft. 
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The ConOps for this test setup was a UAS being controlled by a GCS stationed in southern California, with 
the aircraft deployed under various missions in the San Francisco Bay area. In this test, traffic scenarios 
running on MACS at NASA Ames provided background traffic in Oakland Center (see Section 5 for an 
airspace description). The UAS was flown by an instance of VSCS running on the RGCS at NASA 
Armstrong following a proscribed flight plan. The controllers managed the traffic on MACS running as an 
ERAM display at NASA Ames. The SAA system ran on a machine that was connected to the RGCS via a 
Gateway and provided advisories to the UAS pilot. The pilot vector planned different maneuvers via the 
VSCS traffic display and negotiated appropriate resolution to the predicted conflict with the controller. 
Since the ConOps for the test was chosen to mirror the physical instantiation of the distributed system 
(UAS operating near central California and UAS GCS located near southern California) , no additional 
voice Comm modeling were required to emulate the appropriate latencies. However 93ms uplink delay and 
115ms downlink delay were added to VSCS to emulate latencies associated with data message passing 
between the GCS and aircraft. No winds were modeled for this Test Setup. 

The proposed design of the experiment design includes four levels of advanced Vigilant Spirit display 
information: 

Dl. Advanced display with information only 

D2. Advanced display with information + vector planner 

D3. Advanced display with information + auto resolutions 

D4. Advanced display with information + vector planner + auto resolutions 

For each subject, two scenarios of 38 minutes each were used, and each was replayed twice. During 
repetitions of the scenario two different levels of VSCS displays were tested (randomized over the subject 
pool). Scenario 1 represented a high workload (high traffic density, arrival sequencing for local Terminal 
airspace). Scenario 2 represented a nominal workload (average traffic density, non- arrival push to local 
Terminal). As with Test Setup 1 , these scenarios were developed by the test team controller SME reviewed 
and rated as high and nominal by an independent controller SME. 

The simulated sensor suite provide position data for each intruder aircraft to the SAA algorithm based on 
its aircraft equipment and flight plan type were the same as used for Test Setup 1 (See Figure 10 above): 

1. ADS-B: 

a. Range 80 nm 

b. Altitude +/- 5000 ft 

2. Non-Cooperative: 

a. Range: 6 nmi 

b. Azimuth: +/- 110 degrees (from nose) 

c. Elevation: +/- 20 degrees (from horizontal) 

3. 2. 1.1 Airspace 

The pilot acceptability test setup used the identical airspace as the controller acceptability tests, namely 
Oakland Center sectors 40 and 41 combined. See Figure 1 1 in Section 3.1 .1 .1 for a diagram of the airspace. 

3. 2. 1.2 Scenarios 

Each of the data runs for Test Setup 2 was based upon the CDF Fire Line mission described in Section 
3.1 .1 .2, with changes to the UAS aircraft flight levels and associated background traffic. (See Figure 13 .) 
Table 5 and Table 6 describe the “high” and “low” altitude flight paths of the UAS aircraft in these 
scenarios. 
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Table 5. CDF Fire Line “High” Flight Profile (Route 2) 


Point # 

Fix 

Lat/Long 

Altitude 

1 

Sim Start (VICOR) 

3924.83/-12322.10 

9’ 000 

2 

WEPLO 

3914.26/-12305.63 

9’ 000 

3 

FOWND 

3905 .76/- 12257 .80 

9’ 000 

4 

LAPED 

3903 .70/- 12247 .64 

9’ 000 

5 

WUSHU 

3845.81/-12219.25 

9’ 000 

6 

DALON 

3904.14/-12215.99 

9’ 000 

7 

3912.1 9/- 1 2243 .62 


9’ 000 

8 

3907.78/-12250.30 


9’ 000 

9 

3914.62/-12252.31 


9’ 000 

10 

SABBY 

3917.53/-12302.44 

9’ 000 

11 

HUPTU 

3930.00/-12244.65 

9’ 000 


Table 6. CDF Fire Line “Low” Flight Profile (Route 2) 


Point # 

Fix 

Lat/Long 

Altitude 

1 

Sim Start (VICOR) 

3924.83/-12322.10 

7’ 000 

2 

WEPLO 

3914.26/-12305.63 

7’ 000 

3 

FOWND 

3905.76/-12257.80 

7’ 000 

4 

LAPED 

3903.70/-12247.64 

7’ 000 

5 

WUSHU 

3845.81/-12219.25 

7’ 000 

6 

DALON 

3904.14/-12215.99 

7’ 000 

7 

3912.1 9/- 12243 .62 


7’ 000 

8 

3907.78/-12250.30 


7’ 000 

9 

39 14.62/- 12252.3 1 


7’ 000 

10 

SABBY 

3917.53/-12302.44 

7’ 000 

11 

HUPTU 

3930.00/-12244.65 

7’ 000 
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3.2.2 Test Participants 

The UAS pilot subjects were active Predator B and Global Hawk pilots familiar with GCSs, but no 
experience with the VSCS or the RGCS. The air traffic controller participant (non-subject) was a retired en 
route controller, experienced with using the ERAM display. The test team controller SME trained the 
participant controller on the specific aspects of the ZOA 40/41 sector aircraft flow used for the scenarios. 
Non-subject pilot participants manning the pseudo pilot stations had the appropriate IFR experience to 
communicate with ATC and experience running the MACS pilot software. 

3.2.3 Test Environment Architecture 

The focus of this setup was to investigate the effect of: 

Advanced traffic display elements and tools to enhance pilots’ ability to remain well clear 
Intruder aircraft sensor ranges on the pilots’ ability to remain well clear. 

Figure 18 shows the LVC design for the Pilot Acceptability of SAA Maneuvers test setup. The LVC test 
architecture is nearly identical to Test Setup 1 , with the addition of an Eye Tracker system added to the 
RGCS to track UAS pilot display dwell times and feature focus. 
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Figure 18. IHITL LYC system with components for the Pilot Acceptability of SAA Maneuvers test 
setup (including observer positions) 

3.2.4 Simulation Voice Comm 


The Voice Communication system used for Test Setup 2 was identical to Test Setup 1. Please see Section 
3.1.4 for reference. 


3.3 Test Setup 3 - Controller Acceptability of SAA Advisories and TCAS 
Interoperability 

Test Setup 3 investigated the advisories generated by the NASA Langley’s Stratway+ SAA Concept. This 
concept, relies on the Stratway software prototype originally developed to support tactical resolution 
advisories for manned aircraft, provided SAA informational guidance to the pilot in control in the form of 
“preventive” advisory bands displayed on the ground control station indicating headings and speeds that 
could potentially cause a loss of “well clear” separation with other aircraft. The study was divided into two 
distinct test conditions; the first gathered data to evaluate the interoperability of the SAA advisories and the 
TCAS II algorithm. The second investigated the controller response to the UAS-ATC interactions, subject 
to varying communication delays and varying wind fields. The purpose for dividing the data collection 


32 




























IHITLTest Environment Report - February 17, 2015 


activities was to ensure that the encounters required to test the TCAS alerting could be triggered without 
ATC intervention (which would have been likely). As described above, the two test conditions derived their 
architecture from subsets of System Configuration 2. The first test condition utilized the connection 
between NASA Ames and NASA Langley to connect the Boeing 747 flight simulator at NASA Ames with 
the UAS GCS at NASA Langley. The second condition, focusing on controller response to the delayed 
pilot- ATC communications while exercising self- separation procedures, was conducted stand-alone at 
NASA Langley using only the components shown in the upper box in Figure 3. 

3.3.1 Test Condition 1: SAA/TCAS Interoperability 

The first test condition gathered data to evaluate the interoperability of the SAA Concept advisories and the 
TCAS II algorithm. This study exercised the SAA Concept well clear guidance and examined any possible 
overlap with the TCAS II alerting envelopes. Access to TCAS was provided by an existing NASA Ames 
flight simulator with a instance of TCAS II version 7.0 made available for testing through use of the LVC 
distributed environment. 

3.3.1 .1 Test Methodology 

To conduct the TCAS alerting tests, the flight plan for the B747 aircraft was loaded into the B747 flight 
simulator at NASA Ames and the system at Ames was started. The simulation system at NASA Langley 
(SGT) including the MACS GCS, MACS SimMgr and the MACS monitoring displays was run, configured 
for the ZFW airspace, and seeded with the UAS flight plan profiles that were used to provide encounters 
for SAA and TCAS alerting. Encounters that trigger TCAS resolution advisories (RA) and traffic 
advisories (TA) alerts in the B747 were documented and provided to the researchers for data analysis. The 
initial independent variable of interest was the encounter geometry; the second independent variable was 
the intruder flight path closest point of approach to the unmanned aircraft (0.0, 0.5, and 1 .0 NM). In 
addition, there were three levels of winds (calm, 40 and 80 knots) as headwind, tailwind, and crosswind 
relative to the aircraft. The horizontal missed distance was set at 1 .0 nmi, which should force the self- 
separation alerting to trigger for all three intruder closest point of approach settings. 

Scenarios to test these conditions consisted of scripted encounter situations embedded within a Standard 
Terminal Arrival profile in the DFW airspace to observe/evaluate DAA algorithm performance. Encounter 
conditions invoked loss-of- well-clear geometries. This test condition has no subjects, but was run via input 
files scripted to provide the required encounters. 

3.3.1. 1.1 Airspace 

The simulation airspace was an emulation of Dallas/Ft .Worth Center (ZFW), concentrated an en route 
sector adjacent to Dallas Ft. Worth TRACON and focused on traffic around McKinney National Airport 
(TKI). See Figure 19. This airspace encompasses and was adjacent to a primary arrival stream into DFW. 
When air traffic controller subjects were under test, they performed traffic separation tasks with traffic 
supplied by pseudo pilots and a single UAS pilot. 
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Figure 19. ZFW Airspace near TKI and projected flight path of UAS. 

3.3.1 .1 .2 Scenarios 

For the first test condition, where the flight path of the B747 flight simulator encountered various canned 
UAS aircraft, no specific scenarios were required. These were simple one-on-one encounters to test alerting 
boundaries. Figure 20 shows the path of the B747 in red and the paths of the UAS aircraft in green. Each 
encounter that triggered a TCAS alert was documented. Figure 21 also shows the path of the B747 in red 
and the paths of the UAS aircraft in green. In this case the UAS intruders provided crossing encounters 
along specific points along the B747 flight path. As before, data was gathered for each of the 747/UAS 
encounters. 
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Figure 20. ZFW Airspace head-on collision flight path example. 



Figure 21. ZFW Airspace crossing collision flight path example. 
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3.3.1 .2 Test Participants 

No human subjects were involved in this test condition. Experiment support personnel included: 

• B747 pilots in the flight simulator 

• Trained IFR pilot in control using the MACS GCS with the SAA informative guidance shown on 
an engineering display 

• Research engineers supporting the simulation manager and controlling the LVC gateway station. 

3. 3. 1.3 Test Design 

Figure 22 shows the test environment design for Test Condition 1. The objective of this data collection 
effort was to assess the performance of the Stratway+ SAA capability in encounters with a TCAS II 
equipped aircraft. No human subjects were required, only experiment support personnel consisting of IFR 
pilots and research engineers executing a series of preplanned flight plans for an SAA equipped UAS and a 
TCAS II equipped intruder aircraft. NASA Fangley’s USAR lab at SGT ran the UAS portion of the test, 
with the MACS SimMgr providing the virtual flight plans and the MACS GCS providing the UAS target 
generation and SAA algorithm advisories. The SGT lab ran a MACS GCS and MACS ATC monitoring 
displays all connected to the simulation system via the ADRS. The B747 flight simulator at NASA Ames 
provided the manned intruder aircraft. The B747 was equipped with TCAS II version 7.0 and was run out 
of the CVSRF. The connectivity between the flight simulator and the MACS GCS was provided by the 
DTE infrastructure running out of the DSRF lab at NASA Ames and the Gateway running at SGT. The 
infrastructure contained the HFA messaging system and associated B747 and Gateway Toolboxes that 
converted the messages form the client format into the HFA format. The Gateway provided an interface 
between ADRS and the rest of the simulation environment at SGT. 
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Figure 22. IHITL test infrastructure with components for the Acceptability of SAA Advisories test 
setup. 


3.3.1 .4 Simulation Voice Comm 

Since no test subjects were required for this test condition, a Voice Comm system was not under test. 

3.3.2 Test Condition 2: Controller Acceptability 

The second test condition investigated the controller response to the SAA Concept interaction and delays 
associated with the UAS communication latencies and winds, with the controller as the test subject. This 
included the need to account for ATC-pilot communication latency as well as the GCS-vehicle 
communication latency, using configuration values from the Communications sub-project, to execute pilot- 
initiated SS-requests for trajectory changes. 
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3.3.2. 1 Test Methodology 

This test condition also utilized the ZFW airspace in which a virtual UAS aircraft supplied by the MACS 
GCS interacted with virtual background aircraft supplying the encounters. The air traffic controller subjects 
managed traffic via an en route controller display configured for the ZFW sector adjacent to the TKI 
Terminal area. Additional manned displays for the adjacent “handoff ’ sectors complemented the air traffic 
environment. The virtual GCS contained the DAA algorithm, which supplied the self- separation advisories 
that the UAS pilots used to negotiate maneuvers with the air traffic controller. For each data run, the system 
was loaded with the appropriate scenario files providing virtual aircraft targets to the MACS ATC displays. 
Voice comm delay was provided by the SimPhonics simulation voice comm and audio card hardware. 

During these data runs, three levels of delay were tested, minimal (no added delay above the inherent 
simulation voice comm delay, which is estimated by the manufacturer to be ~100 msec), 100 msec (plus 
the 100 msec inherent delay), 1.0 sec (plus the 100 msec inherent delay). 

The encounters were scripted to occur at 3,000ft and 6,000ft. Based on the scenario, the winds were varied 
between calm and moderate: 

• Calm: 3000’ / direction = 200 / speed = 6 

6000’ / direction = 220 / speed = 7 

• Moderate: 3000’ / direction = 200 / speed = 21 

6000’ / direction = 220 / speed = 23 

Inside a scenario, the wind speed and direction was interpolated based on altitude. 

3. 3. 2. 1.1 Airspace 

The simulation airspace was the same as Test Condition 1 (ZFW). 

3. 3. 2. 1.2 Scenarios 

The controller acceptability of the SAA advisories was tested through 84 traffic encounter scenarios 
embedded in other traffic operations for each subject. The traffic scenarios represented a mix of IFR and 
VFR traffic in the ZFW airspace described above. Each traffic scenario had scripted encounters between 
the UAS aircraft and a manned aircraft. See Figure 23 for an example of one type of encounter (diagram of 
the UAS flight path and encounter flight path shown for illustration). 
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Figure 23. ZFW Airspace near TKI and head-on collision flight path example. 


3. 3. 2. 2 Test Participants 

Air traffic controller subjects were retired en route controllers, experienced with using the ERAM display. 
Controllers were trained on the specific aspects of the ZFW airspace used in the scenarios. Non-subject 
pilot participants manning the pseudo pilot stations had the appropriate IFR experience to communicate 
with ATC and experience running the MACS pilot software. The UAS pilots (non subjects) were 
experienced UAS pilots with prior training on using the MACS GCS engineering display. 

3. 3. 2. 3 Test Design 

Figure 24 shows the simulation architecture for the controller acceptability studies. This study was 
contained solely on site at SGT UAS lab and focused on the controller acceptance of SAA advisories with 
respect to the UAS communication latencies and varying wind fields. As such, this was run as a standalone 
simulation without need for FVC connectivity or infrastructure. A UAS pilot utilized the MACS standalone 
GCS functionality at NASA Fangley’s UAS lab, which provides the UAS simulation functionality. The 
MACS GCS provided a built-in traffic display that provides access to the SAA advisories, which had been 
integrated directly into MACS . The MACS SimMgr provided virtual and constructive manned background 
and intruder aircraft. The MACS pseudo pilot stations provided the target generation of the background 
traffic. The MACS ATC functionality provided the air traffic displays, which included the display for the 
ATC subject, as well as displays for “ghost” controllers that sent and received handoffs for the adjacent 


39 



IHITL Test Environment Report - February 17, 2015 


sectors. As with the TCAS testing, the simulation design included a MACS GCS and a MACS ATC 
monitoring displays. In order to facilitate efficient message passing between the aircraft processes and the 
ATC processes, the architecture included two versions of ADRS, one that communicated to each of the 
pilot stations (and the SimMgr) and one that communicated to each of the MACS ATC displays. These 
instances of ADRS then sent messages to each other as needed. The Voice Comm delay between MACS 
GCS and MACS ATC were implemented by the Simphonics voice system working with the voice comm 
audio cards. 
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Figure 24. IHITL LVC system with components for the Acceptability of SAA Advisories test setup. 


3. 3. 2. 4 Simulation Voice Comm 

The UAS lab at SGT used a Simphonics DIS Voice Communication system that was contained within the 
laboratory. It was configured to add a Voice varying levels of Voice Communication latency to emulate 
potential latencies from a fielded system. 

4 Test Environment Analysis 

The relevancy of the test environment is conditional upon the research data that needs to be collected, so 
hence is dependent on the Test Setups described above. The analyses are divided into two types: 1 .) test 
system latencies, which provides an objective measure of system performance, and 2.) subject matter expert 
feedback, which provides subjective measure of the system acceptability. 


4.1 Test System Latencies 

Latencies in and throughout the LVC test system were measured at three key areas to provide feedback to 
the researchers to ensure the system performed as required. These areas included the latencies between 
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distributed facilities, between the SAA algorithms and the pilot display, and display of traffic data to the 
controllers. 


In order to calculate these latencies, the time on each computer in the LVC system was synchronized to a 
GPS device. Network Time Protocol (NTP) was used for clock synchronization between computer systems. 
NTP provides Coordinated Universal Time (UTC) including scheduled leap second adjustments. The NTP 
uses a hierarchical, semi-layered system of levels of clock sources. Each level of this hierarchy is termed a 
stratum and is assigned a layer number starting with 0 (zero) at the top. The stratum level defines its 
distance from the reference clock and exits to prevent cyclical dependencies in the hierarchy. It is important 
to note that the stratum is not an indication of quality or reliability. To account for any (even minimal) drift 
between two computers, a pair of simple programs were created. The programs “tquery” and “timesvr” 
were used to baseline the time offset between computers residing in two remote facilities of the LVC 
environment. The tquery program calculates the time it takes to send a message to timesvr and then receive 
the reply. The offset is calculated by comparing the average of the time a message tquery send and received 
times to the time the message was originally received by the remote timesvr program. The tquery program 
has the ability to run large sample pools to determine an average and standard deviation on the final 
difference output by the program. 


4.1.1 Facility Latencies 

As described above, the three test setups were derived 
from the two system configurations shown in figures 2 
and 3. For each of the Test Setups, the system was 
emulating an ADS-B traffic system, which provided the 
primary determinate for latency acceptability. The ADS- 
B data transmission and display requirement specified 
by RTCA and the FAA is 5.0 seconds . V1,VU This provided 
a maximum threshold to compare the measured LVC 
system latencies against. There were also other latency 
requirements to consider. Since the pilot display 
guidelines are under investigation, for these tests the 
researchers were looking for less then one second of 
latency between target location and display to the pilot. 
This is well below the ADS-B requirement and serves as 
our benchmark. As the GCS display, DAA algorithm, 
and Communication research matures, the latency 
requirement will be readdressed. 

For Test Setups 1 and 2, the primary background traffic 
and air traffic control displays were stationed at NASA 
Ames, while the UAS Ground Control Station was at 
NASA Armstrong. Figure 25 shows a high-level 
diagram of the facilities used for these tests. To capture 
the overall system latencies, the latencies between each 
of the connected labs were measured. The measurements 
were taken in two ways, first the network latency was 
calculated using the tquery /tserver programs described 
above. This method provides an unimpeded latency 
value. The second method comes from the data recorded 
during a test run of the LVC system. These data are 
captured by the LVC Gateway running at one of the 
facilities and includes delay added by the LVC software. 
Since the LVC software is consistent among the 
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Figure 25. Lab Connectivity for Test 
Setups 1 and 2. 
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Table 7. tquery latency times between NASA Ames 
DSRL Lab and participating IHITL laboratories. 


different labs, once this has been 
recorded between one of the facility 
pairs, the tquery /t server data can be used 
to estimate latencies between other 
facilities for future laboratory 
combinations. 

Table 7 contains the average latencies 
values calculated between the DSRL 
Laboratory at NASA Ames Research 
Center and each of the other participating 
laboratories by the tquery/tserver 
programs. The unimpeded tquery/tserver 
latencies were recorded over a 10-minute 
period with values recorded every 5 
seconds. The observed network latencies 

are very small, with an observed average of 8 ms between Ames and Armstrong and 41 ms between Ames 
and Langley. The difference in latency magnitude between the two setups can be explained by two 
differences in the connections. First, the distance between Ames (which is located in Northern California) 
and Langley (which is in on the East Coast) is much greater then the distance between Ames and 
Armstrong (which is in Southern California). The fastest networks can send data at approximately 1 ms per 
100 miles. So these values are in-line with that measure. Second, the Ames/ Armstrong connection was 
routed over the NASA internal NISN network, while the Ames/Langley connection utilized a Virtual 
Private Network over the Internet. Even with these differences, these results show a very low overall 
latency added by the network. 


Test Name 

Average 

Network 

Latency 

(msec) 

Standard 

Deviation 

(msec) 

Max 

Latency 

(msec) 

Ames to 
Armstrong 

8 

0.3 

13 

Ames to 
Langley (SGT) 

41 

0.1 

41 


Table 8. LVC latency times between NASA Ames 
DSRL Lab and participating IHITL laboratories. 


Table 8 shows the average latencies as 
recorded by the LVC software between 
the DSRL Lab and each of the 
participating IHITL facilities. The data 
was captured during system testing prior 
to the conduct of IHITL and again as a 
normal recording of the IHITL research 
data. The numbers calculated during 
testing show a similar pattern as the 
results from the tquery/tserver testing, 
with a slightly faster connection between 
Ames and Armstrong then Ames and 
Langley. Even considering the 
Maximum observed latencies, the values 
are well within the requirement for 
display and usage. It should be noted that 

the observed DSRL to AFRL latency is below to the 100 ms Voice Comm latency estimate that the system 
was intended to emulate. As we transition to Flight Test, this difference will be discussed with the project 
researchers and if required additional latency will be added to the system as appropriate. 



Average 

LVC 

Latency 

(msec) 

Standard 

Max 

Test Name 

Deviation 

(msec) 

Latency 

(msec) 

Ames to 
Armstrong 

24 

15 

33 

Ames to 

73 

12 

145 

Langley (SGT) 


4.1.2 DAA Advisories 


The calculation of Self Separation advisories by the DAA algorithm and display of the advisories to the 
UAS pilot at the RGCS was of concern to the SSI and HSI researchers, who desired to see a smooth display 
response during trial planning. While the display of the trial planning results was controlled by the VSCS 
software that was outside the LVC infrastructure, the LVC controlled the passing of the DAA alerts and 
advisories between the algorithm and display. In order to facilitate rapid prototyping of algorithm and 
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display interaction, the LVC system was developed to support connection between the algorithms and pilot 
display through the LVC Gateway process. The LVC was required to pass data between the programs at a 
15 Hz rate (~67 msec) to ensure pilot testing of conflict free maneuvers would be smooth. 

4.1 .3 Controller Displays 

While the timing between the facilities is critical to understand whether the LVC infrastructure supports 
relevant testing, the use of MACS as an ATC display was also tested to ensure that data was presented to 
the Air Traffic Control subjects in a timely manner. Though the input data for sensor emulation used the 1- 
second ADS-B update rate, the ATC display was run to emulate present day 12-second update rate of the 
en route workstations. The MACS software was developed to buffer incoming track data from the input 
sources and display that data at the required (12-second) rate. 

4.2 Subject Matter Expert Feedback 

4.2.1 Scenario Development 

As described above, Test Setups 1 and 2 were conducted in an emulation of Oakland Center’s sectors 
40/41 . A retired Oakland Center controller with years of operational experience controlling aircraft in this 
airspace developed the traffic scenarios for Test Setups 1 and 2. Four basic traffic scenarios were created 
based on live traffic recordings. Two scenarios were developed to be high workload (high traffic density, 
arrival sequencing for local Terminal airspace), while the remaining two were developed to be nominal 
workload (average traffic density, non-arrival push to local Terminal). Since high and nominal workloads 
are subjective, these scenarios were also reviewed and rated as high and nominal by a second retired 
Oakland Center controller. 

Similarly, the scenarios developed for Test Setup 3 were based on live air traffic data recordings of the 
Dallas Ft Worth Center and Terminal airspaces. These recordings were used by the SSI research team to 
populate a set of background traffic in which a candidate UAS aircraft flight path was inserted. 

4.2.2 Controller Subject Evaluation 

As part of the conduct of Test Setup 1 , the test subject was asked to evaluate the realism and relevance of 
the environment and traffic displays. The results are captured here. 

4.2.2. 1 Post Scenario Questions 

1 .) “The traffic density in my sector was realistic relative to current-day operations:” 
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■ Scenario 1 ■ Scenario 2 ■ Scenario 3 ■ Scenario 4 
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Much less Somewhat less Typical Somewhat Much busier 

busy than busy than busier than than normal 

normal normal normal 


The result of this question is interesting. Scenarios 1 and 4 were scripted to be high workload, but both had 
the most responses of “typical”, while the nominal workload scenarios (2 and 3) were rating busier. 
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2.) “As compared with normal real-world operations, the number of the encounters in this trial was:” 


■ Scenario 1 ■ Scenario 2 ■ Scenario 3 ■ Scenario 4 


12 

10 

8 

6 



i 


Much less Somewhat Typical 
frequent less frequent 


Somewhat Much more 
more frequent 
frequent 


The result of this question mirror results from the previous question. Though scenarios 1 and 4 were 
scripted to be high workload from an ATC point of view, the subjects had rated the number of encounters, 
and the perceived traffic density that accompanies those scenarios as lower. 


3.) “As compared with normal real-world operations, the complexity of the encounters in this trial was:” 


■ Scenario 1 ■ Scenario 2 ■ Scenario 3 ■ Scenario 4 

12 
10 
8 
6 
4 
2 
0 

Much easier 
to detect and 
resolve 


These results also support the perceived lower traffic load results for Scenario 1 and 4 from the first 
question. Not only were the number of encounters perceived to be lower, but the complexity was also rated 
lower then scenarios 2 and 3 . 



Somewhat Neither Somewhat Much more 
easier to easier nor more difficult difficult to 
detect and more difficult to detect and detect and 
resolve to detect and resolve resolve 
resolve 
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4.) “As compared with normal real-world operations, how would you rate your workload during this 
scenario?” 

■ Scenario 1 ■ Scenario 2 ■ Scenario 3 ■ Scenario 4 

10 
9 
8 
7 
6 
5 
4 
3 
2 
1 
0 

Much lower 
than normal 

This result substantiates the previous results. The traffic load was perceived to be lower for Scenarios 1 and 
4, as was the workload associated with handling the traffic. This underscores the need to include the study 
of controller workload when assessing UAS encounters with air traffic. 



Somewhat Typical Somewhat Much higher 

lower than higher than than normal 

normal normal 


5.) “ Did the workload and complexity of the scenario impact your ability to provide additional services 
( direct routings, traffic advisories, etc.)?” 


■ Scenario 1 ■ Scenario 2 ■ Scenario 3 ■ Scenario 4 


9 



on additional impact on provide a impact on provide 

services additional typical additional additional 

services degree of services services 
addition 
services 

This result substantiates the previous results in terms of a lower workload for Scenarios 1 and 4. This 
underscores the criticality of supporting controllers with handling encounters with UAS . It also begins to 
demonstrate potential impacts that controller skill levels and familiarity with the sector can have on the 
ability to provide additional services. 
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6. ) “The flow of traffic in my sector was representative of a low altitude en route sector:” 

■ Scenario 1 ■ Scenario 2 ■ Scenario 3 ■ Scenario 4 

16 
14 
12 
10 
8 
6 
4 
2 
0 

uncertain no yes 

This question addresses the heart of this study. The vast majority of controllers felt that the scenario design 
and traffic behavior provided a realistic representation of a real world low altitude en route sector. 

7. ) “Did you notice any ghost controller, pilot, or software behavior in the simulation that did not 
conform to your expectations or that was unlike behavior in the real world?” 




Controller responses to this question focused on actions taken by the UAS pilot and pseudo pilots, and their 
actions in response to either observed/detected traffic or traffic called out by the controller. The responses 
provided by those who answered 44 Yes” are provided below, and indicate a higher than normal number of 
pilot requests for deviations around traffic. This is not unexpected given the higher than average number of 
traffic encounters built into the simulation scenario. 
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“A few too many pilots asking to deviate around known traffic; can we rely on UAS reported altitude of 
merging aircraft? ” 

“My [operational] experience was pilots rarely spotted traffic very early and almost never asked for turns 
around the traffic. They usually spot the traffic and just separate themselves with altitude or laterally. If the 
UAS detects traffic I would think they would trust their equipment regarding altitude or headings— or— 
they would ask ATC about the traffic, then respond appropriately A 

“Most aircraft that I called traffic or had traffic turned— not really true in my experience . Most stay the 
course especially when they get them in sight A 

“Some handoffs were late, which may have been a result of a lag in the A/C sim start or I didn't notice 
them until the AC was already in my airspace. The real world scenario would have a controller calling for 
a handoff. ” 

A.2.2.2 Post Simulation Questions 

The controller evaluations had only one post relevant post simulation question pertaining to the relevancy 
of the LVC environment: 

“As compared with air traffic control displays used operationally, the ATC display environment used 
during the simulation:” 
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Had major 
display issues 
that prevented 
me from 
performing 
routine tasks. 


Had minor 
display issues 
that detracted 
from my ability 
for control 
traffic 


Was adequate 
for controlling 
air traffic 


Was a good 
emulation of a 
controller 
workstation, 
with only minor 
differences 


Was a realistic 
emulation of a 
controller 
workstation 


Controller responses to this question overwhelmingly indicated that the software simulation was, at a 
minimum, “adequate” for the assigned control task. The comments below were provided by controllers 
who indicated that there were minor display issues that detracted from their ability to control traffic. 


“I had an issue with my keyboard a couple of times when taking handoffs - I had to resort to my trackball.” 
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“All data blocks began flashing at once. I couldn’t make any entries while that was occurring (less than 1 
minute). Also, I had trouble moving data blocks with the slewball.” 

“As noted, flashing cursor,jumping targets around Salsalito, and one dropped data block. Pilots/remote 
were excellent! Software simulation was excellent. UAS was great; nice addition.” 


4.2.3 Pilot Subject Evaluation 

No questions regarding the realism of the simulation were included in the UAS pilot surveys. The primary 
reason is that the pilot subjects were evaluating prototype displays that were designed to understand levels 
of display information, not necessarily a realistic display. There were, however, remarks made by the pilot 
participants during the runtime of the study that were relevant to the simulation’s fidelity. Pilots often 
commented on the maturity of the research ground control station. Multiple pilots stated that the software 
suite provided by Vigilant Spirit Control Station presented them with a more flexible, intuitive and robust 
system than they typically fly with. Pilots also consistently commended the realistic environment provided 
by the controller and pseudo-pilots. Several pilots remarked that the chatter over the frequency reached a 
realistic density, which often forced them to make a realistic decision: do they postpone a maneuver around 
traffic until they can find an opening on the frequency, or do they start the maneuver without prior 
approval? Pilots also commented on the professional nature of the communications from the controller and 
pseudo pilots, noting that all the parties sounded experienced in flying and controlling aircraft. 

While the remarks from the pilot participants were largely positive, there were several comments during 
simulation that were more critical. For instance, pilots felt that the controllers in this portion of the study 
were too passive, either missing critical encounters between the UAS and other aircraft, or never refusing a 
request by the pilot to maneuver around traffic. Pilots also felt that the number of encounters they 
experienced during the course of a single trial was far higher than they would expect in the real world. 
While the number of generated encounters and the behavior of the controller were purposefully scripted to 
accommodate research goals, it should be noted that these decisions impact the level of realism experienced 
by participants. 

5 Environment Relevancy 

Based on both controller and pilot feedback as well as the methodical approach that was used to construct 
the scenarios using recorded traffic as the baseline, we are satisfied that the simulation met the requirement 
to be “realistic”. Controller-identified issues primarily focused on equipment or pseudo pilot procedures. 
Equipment issues were, for the most part, isolated anomalies. However, several issues were identified by 
our team that will be addressed in future versions of MACS ERAM software in order to provide console 
capabilities that controllers are used to having access to. 

Feedback on pseudo pilot deviations for either pilot observed or controller reported traffic was the primary 
criticism from both controller and UAS pilot test subjects. The construction of the scenario encounters 
between the UAS and other traffic was intentionally designed to place aircraft in close proximity if they did 
not deviate in order to provide data for the researchers on the performance of the self- separation algorithms 
and UAS operator displays. The requirement for a higher than normal frequency of close proximity 
encounters was expected to generate responses such as these, but were a necessary part of the scenario to 
support the research. 

Based on the feedback from the controllers and pilots and since the relevancy of the environment depends 
on the data to be collected to support the research, the LVC test environment was deemed relevant. 
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Appendix A: Acronyms 


ADRS 

Aeronautical Data link and Radar Simulator 

ADS-B 

Automatic Dependent Surveillance-Broadcast 

AFRC 

Armstrong Flight Research Center 

AFRL 

Air Force Research Lab 

APG 

Annual Performance Goal 

ARC 

Ames Research Center 

ATC 

Air Traffic Control 

ATWIT 

Air Traffic Workload Input Technique 

CA 

Collision Avoidance 

CDF 

California Department of Forestry 

CDTI 

Cockpit Display of Traffic Information 

CNPC 

Comman and Non-Payload Control 

ConOps 

Concept of Operations 

COTS 

Commercial Off-the-Shelf 

CSD 

Cockpit Situation Display (A specific type of CDTI) 

CVSRF 

Crew Vehicle Systems Research Facility 

DFW 

Dallas/Ft. Worth Airport 

DSRL 

Distributed System Research Laboratory at NASA Ames 

DTE 

Distributed Test Environment 

ERAM 

En Route Automation Modernization 

FAA 

Federal Aviation Administration 

FDDRL 

Flight Deck Display Research Laboratory at NASA Ames 

FMS 

Flight Management System 

GCS 

Ground Control Station 

GCS-TD 

Ground Control Station - Traffic Display 
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GOTS 

HLA 

HSI 

HITL 

ICD 

IEEE 

IFR 

IHITL 

IT&E 

LaRC 

LOS 

LVC 

MACS 

MOPS 

NASA 

ORD 

RAIF 

RFDP 

RGCS 

RTCA 

RUMS 

SA 

SAA 

SC-228 

SGT 

SME 

SSI 

SS 

STARS 


Government Off-the- Shelf 

High Level Architecture 

Human Systems Integration 

Human-In-The-Loop 

Interface Control Document 

Institute of Electrical and Electronics Engineers 

Instrument Flight Rules 

Integrated Human in the Loop simulation test series 
Integrated Test and Evaluation 
Langley Research Center 
Loss of Separation 

Live Virtual Constructive network system 

Multi-Aircraft Control System 

Minimum Operation Performance Standards 

National Aeronautics and Space Administration 

Objectives and Requirements Document 

Research Aircraft Integration Facility at NASA Armstrong 

Research Flight Data Processor 

Research Ground Control Station 

Radio Technical Commission for Aeronautics 

Remote User Monitoring System 

Separation Assurance 

Sense and Avoid 

RTCA Special Committee 228 

Stinger Gaffarian Technologies outside of NASA Langley 
Subject Matter Expert 

Separation Assurance-Sense and Avoid Interoperability 
Self Separation 

Standard Terminal Automation Replacement System 
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TC 

Test Conductor 

TCAS 

Traffic Collision Avoidance System 

TD/MD 

Test Director/ Mission Director (interchangeable) 

TKI 

McKinney National Airport 

TRACON 

Terminal Radar Approach Control 

UAS-NAS 

Unmanned Aircraft System in the National Airspace System 

UASR 

UAS Sense and Avoid Research 

UAV 

Unmanned Aerial Vehicle 

VFR 

Visual Flight Rules 

VSCS 

Vigilant Spirit Control Station 

WCV 

Well Clear Volume 

ZFW 

Dallas/Ft. Worth Center 

ZOA 

Oakland Center 
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